Generation of domain-specific models in networked system

ABSTRACT

The present disclosure is generally directed to the generation of domain-specific, voice-activated systems in interconnected networks. The system can receive input signals that are detected at a client device. The input signals can be voice-based input signals, text-based input signals, image-based input signals, or other type of input signals. Based on the input signals, the system can select domain-specific knowledge graphs and generate responses based on the selected knowledge graph.

BACKGROUND

In a networked environment, such as the Internet or other networks,first-party content providers can provide information for publicpresentation on resources such as web pages, documents, applications,and other resources. The first-party content can include text, video,and audio information provided by the first-party content providers. Auser accessing the resources may request or wish to receive additionalinformation about information related to the resources. To view orreceive the additional information, the user needs to leave thecurrently viewed resource.

SUMMARY

According to at least one aspect of the disclosure, a system to generatenatural language processing models in a networked system can include adata processing system. The data processing system can include one ormore processors and memory. The one or more processors can execute anatural language processor component and a digital component selector.The data processing system can receive, by the natural languageprocessor component and via an interface of the data processing system,an input audio signal detected by a sensor at a first client device. Thedata processing system can parse, by the natural language processorcomponent, the input audio signal to identify a first search request inthe first search request in the input audio signal. The data processingsystem can select, by a digital component selector executed by the dataprocessing system, a search result based on at least the first searchrequest. The data processing system can select, by the digital componentselector, a digital component based on the first search request. Thedigital component can include a link to a data source. The data sourcecan include a plurality of entities. The digital component can includean input interface to request responses based on the knowledge graph.

The data processing system can transmit, by the interface, the digitalcomponent including the link to the with the data source and associatedwith the knowledge graph to the first client device with the searchresult. The data processing system can receive, by the interface and viathe input interface of the digital component rendered by the clientdevice, a second search request. The data processing system can select,by the digital component selector and based on the knowledge graph, aresponse based to the second search request. The data processing systemcan transmit, by the interface, the response to the first client deviceto render the response within the digital component.

According to at least one aspect of the disclosure, a method to generatenatural language processing models in a networked system can includereceiving, by a natural language processor component executed by a dataprocessing system and via an interface of the data processing system, aninput audio signal detected by a sensor at a first client device. Themethod can include parsing, by the natural language processor component,the input audio signal to identify a first search request in the firstsearch request in the input audio signal. The method can includeselecting, by a digital component selector executed by the dataprocessing system, a search result based on the at least the firstsearch request. The method can include selecting, by the digitalcomponent selector, a digital component based on the first searchrequest. The digital component can be associated with a data source anda knowledge graph based on the data source. The method can includetransmitting, by the digital component selector, the digital componentassociated with the data source and the knowledge graph based on thedata source to the first client device with the search result.

According to at least one aspect of the disclosure, a system to generatenatural language processing models in a networked system can include adata processing system. The data processing system can include one ormore processor and memory. The one or more processors can execute adigital component selector and a natural language processor component.The data processing system can receive, by the digital componentselector, a content request from a client device. The data processingsystem can select, by the digital component selector, a digitalcomponent based on the content request. The digital component can beassociated with a data source and a knowledge graph based on the datasource. The data processing system can transmit, by the digitalcomponent selector, the digital component to the client computing deviceto render the digital component in a content slot. The data processingsystem can receive, by the natural language processor component and viaan interface of the data processing system, an input audio signaldetected by a sensor at the client device. The data processing systemcan parse, by the natural language processor component, the input audiosignal to identify a request in the input audio signal. The dataprocessing system can select, by the natural language processorcomponent, a response to the request based on the knowledge graph. Thedata processing system can transmit, by the interface, the response tothe client computing device.

According to at least one aspect of the disclosure, a method to generatenatural language processing models in a networked system can includereceiving, by a digital component selector executed by a data processingsystem, a content request from a client computing device. The method caninclude selecting, by the digital component selector, a digitalcomponent based on the content request. The digital component can beassociated with a data source and a knowledge graph based on the datasource. The method can include transmitting, by the digital componentselector, the digital component to the client computing device to renderthe digital component in a content slot. The method can includereceiving, by the natural language processor, an input audio signaldetected by a sensor at the client device. The method can includeparsing, by the natural language processor component, the input audiosignal to identify a request in the input audio signal. The method caninclude selecting, by the natural language processor component, aresponse to the request based on the knowledge graph. The method caninclude transmitting, by the interface, the response to the clientcomputing device.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations andprovide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component may be labeled inevery drawing. In the drawings:

FIG. 1 illustrates an example system to generate responses based ondomain-specific natural language models in a networked computerenvironment, in accordance with an example of the present disclosure.

FIG. 2 illustrates a block diagram of the system illustrated in FIG. 1generating and using example knowledge graphs, in accordance with anexample of the present disclosure.

FIGS. 3 and 4 illustrate block diagrams of example methods to generateresponses based on domain-specific natural language processing models ina networked system, in accordance with an example of the presentdisclosure.

FIG. 5 illustrates a block diagram of an example computer system thatcan be used in the system illustrated in FIG. 1, in accordance with anexample of the present disclosure.

DETAILED DESCRIPTION

Following below are more detailed descriptions of various conceptsrelated to, and implementations of, methods, apparatuses, and systems togenerate voice-activated data flows in interconnected network. Thevarious concepts introduced above and discussed in greater detail belowmay be implemented in any of numerous ways.

The present disclosure is generally directed to the generation ofdomain-specific, voice-activated systems in interconnected networks. Thesystem can receive input signals that are detected at a client device.The input signals can be voice-based input signals, text-based inputsignals, image-based input signals, or other type of input signals. Theinput signals can include requests, such as search requests. Because ofthe conversational nature of the input signals, the requests can bevague, broad, or lack context. The system can use the domain-specificknowledge graphs to generate responses to the requests. Interfaces tothe domain-specific knowledge graphs can be incorporated into digitalcomponents that are provided in response to search requests or providedwith first party content on a web page.

Systems and methods of the present technical solution enable a reductionin reducing network bandwidth by reducing the number of networktransmissions required to complete voice-based requests. The presentsolution can enable the digital assistant to select specific responsesto unclear requests, which can reduce the number of follow up inputaudio signals required to complete a request.

FIG. 1 illustrates an example system 100 to generate responses based ondomain-specific natural language models in a networked computerenvironment. The system 100 can include a data processing system 102.The data processing system 102 can communicate with one or more digitalcomponent provider devices 106 (e.g., content provider devices) orclient computing devices 104 via a network 105.

The system 100 can include one or more networks 105. The network 105 caninclude computer networks such as the Internet, local, wide, metro, orother area networks, intranets, satellite networks, other communicationnetworks such as voice or data mobile telephone networks, or anycombination thereof.

The data processing system 102 and client computing device 104 canaccess digital components and data sources 135 via the network 105. Thenetwork 105 can be used to access data sources such as web pages,websites, domains (e.g., a collection of web pages), or uniform resourcelocators. The digital components can be presented, output, rendered, ordisplayed on at least one computing device 104, such as a laptop,desktop, tablet, digital assistant, personal digital assistant,smartwatch, wearable device, smart phone, portable computer, or speaker.For example, via the network 105 a user of the client computing device104 can access a website (an example data source 135) provided bydigital component provider devices 106. The website can include one ormore digital components, such as first- and third-party content.

The network 105 can include or constitute a display network, e.g., asubset of information resources available on the internet that areassociated with a content placement or search engine results system orthat are eligible to include third-party digital components. The network105 can be used by the data processing system 102 to access informationresources such as web pages, websites, domain names, or uniform resourcelocators that can be presented, output, rendered, or displayed by theclient computing device 104.

The network 105 may be any type or form of network and may include anyof the following: a point-to-point network, a broadcast network, a widearea network, a local area network, a telecommunications network, a datacommunication network, a computer network, an ATM (Asynchronous TransferMode) network, a SONET (Synchronous Optical Network) network, a SDH(Synchronous Digital Hierarchy) network, a wireless network, and awireline network. The network 105 may include a wireless link, such asan infrared channel or satellite band. The topology of the network 105may include a bus, star, or ring network topology. The network mayinclude mobile telephone networks using any protocol or protocols usedto communicate among mobile devices, including advanced mobile phoneprotocol (“AMPS”), time division multiple access (“TDMA”), code-divisionmultiple access (“CDMA”), global system for mobile communication(“GSM”), general packet radio services (“GPRS”), or universal mobiletelecommunications system (“UMTS”). Different types of data may betransmitted via different protocols, or the same types of data may betransmitted via different protocols.

The system 100 can include at least one data processing system 102. Thedata processing system 102 can include at least one logic device such asa computing device having a processor to communicate via the network105, for example, with the computing device 104 or the digital componentprovider device 106. The data processing system 102 can include at leastone computation resource, server, processor, or memory. For example, thedata processing system 102 can include a plurality of computationresources or servers located in at least one data center. The dataprocessing system 102 can include multiple, logically-grouped serversand facilitate distributed computing techniques. The logical group ofservers may be referred to as a data center, server farm, or a machinefarm. The servers can also be geographically dispersed. A data center ormachine farm may be administered as a single entity, or the machine farmcan include a plurality of machine farms. The servers within eachmachine farm can be heterogeneous; one or more of the servers ormachines can operate according to one or more type of operating systemplatform.

Servers in the machine farm can be stored in high-density rack systems,along with associated storage systems, and located in an enterprise datacenter. For example, consolidating the servers in this way may improvesystem manageability, data security, the physical security of thesystem, and system performance by locating servers and high-performancestorage systems on localized high-performance networks. Centralizationof all or some of the data processing system 102 components, includingservers and storage systems, and coupling them with advanced systemmanagement tools allows more efficient use of server resources, whichsaves power and processing requirements and reduces bandwidth usage.

The client computing device 104 can include, execute, interface, orotherwise communicate with one or more of at least one local digitalassistant 134, at least one sensor 138, at least one transducer 140, atleast one audio driver 142, or at least one display 144. The clientcomputing device 104 can interface with one or more interfaces, such asa graphical or physical keyword.

The sensor 138 can include, for example, a camera, an ambient lightsensor, proximity sensor, temperature sensor, accelerometer, gyroscope,motion detector, GPS sensor, location sensor, microphone, video, imagedetection, or touch sensor. The transducer 140 can include or be part ofa speaker or a microphone. The audio driver 142 can provide a softwareinterface to the hardware transducer 140. The audio driver 142 canexecute the audio file or other instructions provided by the dataprocessing system 102 to control the transducer 140 to generate acorresponding acoustic wave or sound wave. The display 144 can includeone or more hardware or software components configured to provide avisual indication or optical output, such as a light emitting diode,organic light emitting diode, liquid crystal display, laser, or display.

The client computing device 104 may or may not include a display. Forexample, the client computing device 104 may include limited types ofuser interfaces, such as a microphone and speaker (e.g., the clientcomputing device 104 can include a voice-drive or audio-basedinterface). The client computing device 104 can be a speaker-baseddigital assistant. The primary user interface of the computing device104 can include a microphone and speaker.

The client computing device 104 can include, execute, interface, orotherwise communicate with the local digital assistant 134. The localdigital assistant 134 can detect input signals, such as audio inputsignals, at the client computing device 104. The input signals caninclude requests or search requests. The local digital assistance 134can be an instance of the remote digital assistance component 112executed at the data processing system 102 or can perform any of thefunctions of the remote digital assistance component 112.

The local digital assistant 134 can filter out one or more terms ormodify the terms prior to transmitting the terms as data to the dataprocessing system 102 (e.g., remote digital assistant component 112) forfurther processing. The local digital assistant 134 can convert theanalog audio signals detected by the transducer 140 into a digital audiosignal and transmit one or more data packets carrying the digital audiosignal to the data processing system 102 via the network 105. The localdigital assistant 134 can transmit data packets carrying some or all ofthe input audio signal responsive to detecting an instruction to performsuch transmission. The instruction can include, for example, a triggerkeyword or other keyword or approval to transmit data packets comprisingthe input audio signal to the data processing system 102.

The local digital assistant 134 can interface with one or more of theknowledge graphs 126 hosted or generated by the data processing system102. The local digital assistant 134 can provide or render an interfaceto the knowledge graphs 126. For example, the local digital assistant134 can receive input signals that are transmitted to the dataprocessing system 102. The remote digital component assistant 112 candetermine a response to the request based at least on the knowledgegraph 126. The local digital assistant 134 can interface with theknowledge graphs 126 to provide results or responses to requests parsedfrom input signals.

The local digital assistant 134 can perform pre-filtering orpre-processing on the input audio signal to remove certain frequenciesof audio. The pre-filtering can include filters such as a low-passfilter, high-pass filter, or a bandpass filter. The filters can beapplied in the frequency domain. The filters can be applied usingdigital signal processing techniques. The filter can be configured tokeep frequencies that correspond to a human voice or human speech, whileeliminating frequencies that fall outside the typical frequencies ofhuman speech. For example, a bandpass filter can be configured to removefrequencies below a first threshold (e.g., 70 Hz, 75 Hz, 80 Hz, 85 Hz,90 Hz, 95 Hz, 100 Hz, or 105 Hz) and above a second threshold (e.g., 200Hz, 205 Hz, 210 Hz, 225 Hz, 235 Hz, 245 Hz, or 255 Hz). Applying abandpass filter can reduce computing resource utilization in downstreamprocessing. The local digital assistant 134 on the computing device 104can apply the bandpass filter prior to transmitting the input audiosignal to the data processing system 102, thereby reducing networkbandwidth utilization. However, based on the computing resourcesavailable to the computing device 104 and the available networkbandwidth, it may be more efficient to provide the input audio signal tothe data processing system 102 to allow the data processing system 102to perform the filtering.

The local digital assistant 134 can apply additional pre-processing orpre-filtering techniques such as noise reduction techniques to reduceambient noise levels that can interfere with the natural languageprocessor. Noise reduction techniques can improve accuracy and speed ofthe natural language processor, thereby improving the performance of thedata processing system 102, and manage rendering of a graphical userinterface provided via the display 144.

The client computing device 104 can be associated with an end user thatenters voice queries as audio input into the client computing device 104(via the sensor 138 or transducer 140) and receives audio (or other)output to present, display, or render to the end user of the clientcomputing device 104 from the data processing system 102 or digitalcomponent provider device 106.

The digital component can include a computer-generated voice that can beprovided from the data processing system 102 or digital componentprovider device 106 to the client computing device 104. The clientcomputing device 104 can render the computer-generated voice to the enduser via the transducer 140 (e.g., a speaker). The computer-generatedvoice can include recordings from a real person or computer-generatedlanguage. The client computing device 104 can provide visual output viaa display device 144 communicatively coupled to the computing device104. The client computing device 104 can receive queries from the enduser via a keyboard. The query can be a request or a search request.

The client computing device 104 can receive an input audio signaldetected by a sensor 138 (e.g., microphone) of the computing device 104.The input audio signal or other form of input signal can include, forexample, a query, question, command, instructions, request, searchrequest, or other statement provided in a spoken language.

The client computing device 104 can include, execute, or be referred toas a digital assistant device. The digital assistant device can includeone or more components of the computing device 104. The digitalassistant device can include a graphics driver that can receive displayoutput from the data processing system 102 and render the display outputon display 132. The graphics driver can include hardware or softwarecomponents that control or enhance how graphics or visual output isdisplayed on the display 144. The graphics driver can include, forexample, a program that controls how the graphic components work withthe rest of the computing device 104 (or digital assistant). The localdigital assistant 134 can filter the input audio signal to create afiltered input audio signal, convert the filtered input audio signal todata packets, and transmit the data packets to a data processing systemcomprising one or more processors and memory.

The digital assistant device can include an audio driver 142 and aspeaker component (e.g., transducer 140). The pre-processor component140 can receive an indication of the display output and instruct theaudio driver 142 to generate an output audio signal to cause the speakercomponent (e.g., transducer 140) to transmit an audio outputcorresponding to the indication of the display output.

The system 100 can include, access, or otherwise interact with at leastdigital component provider device 106. The digital component providerdevice 106 can include one or more servers that can provide digitalcomponents to the client computing device 104 or data processing system102. The digital component provider device 106 can provide or beassociated with data sources 135. The data sources 135 can be websitesor landing pages. The digital components provided by the digitalcomponent provider device 106 can be associated with a data source 135provided by the digital component provider device 106. For example, thedigital component can be third-party content that includes a link to adata source 135, such as a landing page, provided by the digitalcomponent provider device 106.

The digital component provider device 106 or components thereof can beintegrated with the data processing system 102 or executed at leastpartially by the data processing system 102. The digital componentprovider device 106 can include at least one logic device such as acomputing device having a processor to communicate via the network 105,for example, with the computing device 104, the data processing system102, or the digital component provider device 106. The digital componentprovider device 106 can include at least one computation resource,server, processor, or memory. For example, the digital componentprovider device 106 can include a plurality of computation resources orservers located in at least one data center.

A digital component provider device 106 can provide audio, visual, ormultimedia-based digital components for presentation by the clientcomputing device 104 as output digital components or visual outputdigital components. The digital components can be websites, web pages,applications, text-based content, audio-based content, video-basedcontent, other digital documents, or any combination thereof. Thedigital component can be or include digital content. The digitalcomponent can be or include a digital object. A digital component caninclude a plurality of digital content items or other digitalcomponents. For example, a digital component can be a website thatincludes other digital components such as ads or content from thirdparties. A digital component can include an instance of the localdigital assistant 134 or cause the client computing device 104 toexecute an instance of the local digital assistant 134.

The digital component provider device 106 can provide the digitalcomponents to the client computing device 104 via the network 105 andbypass the data processing system 102. The digital component providerdevice 106 can provide the digital component to the client computingdevice 104 via the network 105 and data processing system 102. Forexample, the digital component provider device 106 can provide thedigital components to the data processing system 102, which can storethe digital components and provide the digital components to the clientcomputing device 104 when requested by the client computing device 104.The digital components can be from a data source 135. The data sourcecan be a server hosting a webpage, landing page, or other content.

The data processing system 102 can include at least one computationresource or server. The data processing system 102 can include,interface, or otherwise communicate with at least one interface 110. Thedata processing system 102 can include, interface, or otherwisecommunicate with at least one remote digital assistant component 112.The remote digital assistant component 112 can include, interface, orotherwise communicate with at least one NLP component 114, and at leastone domain processor 117. The data processing system 102 can include,interface, or otherwise communicate with at least one digital componentselector 120. The data processing system 102 can include, interface, orotherwise communicate with at least one data repository 124. The atleast one data repository 124 can include or store, in one or more datastructures or databases, knowledge graphs 126 and content data 132.

The components of the data processing system 102 can each include atleast one processing unit or other logic device such as a programmablelogic array engine or module configured to communicate with the databaserepository or database 124. The components of the data processing system102 can be separate components, a single component, or part of multipledata processing systems 102. The system 100 and its components, such asa data processing system 102, can include hardware elements, such as oneor more processors, logic devices, or circuits.

The data processing system 102 can include an interface 110. Theinterface 110 can be configured, constructed, or operational to receiveand transmit information using, for example, data packets. The interface110 can receive and transmit information using one or more protocols,such as a network protocol. The interface 110 can include a hardwareinterface, software interface, wired interface, or wireless interface.The interface 110 can facilitate translating or formatting data from oneformat to another format. For example, the interface 110 can include anapplication programming interface (“API”) that includes definitions forcommunicating between various components, such as software components.

The remote digital assistant component 112 of the data processing system102 can execute or run an NLP component 114 to receive or obtain thedata packets including the input signal. The input signal can includeinput audio signals detected by the sensor 138 of the computing device104 or other input signals, such as entered text. For example, the inputsignal can include text input into the client computing device 104 by auser via a keyboard or other text entry system. The data packets canprovide a digital file. The NLP component 114 can receive or obtain thedigital file or data packets comprising the input signal and parse theinput signal. For example, the NLP component 114 can enable forinteractions between a human and a computer. The NLP component 114 canbe configured with techniques for converting the input signal to textand understanding natural language to enable the data processing system102 to derive meaning from human or natural language input.

The NLP component 114 can include or be configured with techniques basedon machine learning, such as statistical machine learning. The NLPcomponent 114 can utilize decision trees, statistical models, orprobabilistic models to parse the input audio signal. The NLP component114 can perform, for example, functions such as named entity recognition(e.g., given a stream of text, determine which items in the text map tonames, such as people or places, and what the type of each such name is,such as a person, location (e.g., “home”), or organization), naturallanguage generation (e.g., convert information from computer databasesor semantic intents into understandable human language), naturallanguage understanding (e.g., convert text into more formalrepresentations such as first-order logic structures that a computermodule can manipulate), machine translation (e.g., automaticallytranslate text from one human language to another), morphologicalsegmentation (e.g., separating words into individual morphemes andidentify the class of the morphemes, which can be challenging based onthe complexity of the morphology or structure of the words of thelanguage being considered), question answering (e.g., determining ananswer to a human-language question, which can be specific oropen-ended), or semantic processing (e.g., processing that can occurafter identifying a word and encoding its meaning in order to relate theidentified word to other words with similar meanings).

The NLP component 114 can convert the input audio signal into recognizedtext by comparing the input signal against a stored, representative setof audio waveforms (e.g., in the data repository 124) and choosing theclosest matches. The set of audio waveforms can be stored in the datarepository 124 or other database accessible to the data processingsystem 102. The representative waveforms can be generated across a largeset of users, and then may be augmented with speech samples from theuser. After the audio signal is converted into recognized text, the NLPcomponent 114 matches the text to words that are associated, forexample, via training across users or through manual specification, withactions that the data processing system 102 can serve. The NLP component114 can convert image or video input into text or digital files. The NLPcomponent 114 can process, analyze, or interpret image or video input toperform actions, generate requests, or select or identify datastructures.

The NLP component 114 can obtain the input signal. From the inputsignal, the NLP component 114 can identify at least one request. Therequest can indicate intent or digital components or can be a searchrequest. The request can be an expressly stated request for information.For example, the request can be the question “what colors does car modelX come in?” The intent can also be derived or not expressly stated. Forexample, in the input signal “car model X 2018,” the input signal doesnot expressly state that the user wants information about the 2018 carmodel X, but the NLP component 114 can derive the intent.

The NLP component 114 can parse the input signal to identify, determine,retrieve, or otherwise obtain the request from the input signal. Forinstance, the NLP component 114 can apply a semantic processingtechnique to the input signal to identify a search request in the inputsignal.

The data processing system 102 can include or interface with an instanceof the domain processor 117. The domain processor 117 can be any script,file, program, application, set of instructions, or computer-executablecode, that is configured to enable a computing device on which thedomain processor 117 is executed to generate knowledge graphs 126.

The domain processor 117 can generate knowledge graphs 126 that aredomain-specific. For example, the domain processor 117 can generateknowledge graphs 126 for different specific websites, domains, datacollections, other data sources 135. The domain processor 117 cangenerate a knowledge graph 126 for each of the digital components thatthe data processing system 102 receives from the digital componentprovider device 106. The domain processor 117 can save the knowledgegraphs 126 into the data repository 124. The domain processor 117 cansave the knowledge graphs 126 in a relational database.

The data sources 135, such as the websites under a specific domain, caninclude terms, phrases, or other data that can generally be referred toas entities. The knowledge graphs 126 for a given data source 135 caninclude nodes that represent the entities in the data source 135.

A knowledge graph 126 can include edges or links that connect relatednodes in the knowledge graph 126. The edges can represent relationshipsbetween the entities. For example, two nodes linked by an edge canindicate that the entities represented by the nodes are related. Thedomain processor 117 can assign weights to the edges. The weights canindicate a degree of the relationship between the nodes connected by thelink. For example, an edge with a higher weight can indicate that thetwo entities represented by the nodes are more related to one anotherthan two entities that are connected by an edge with a relatively lowerweight. An edge can represent a semantic relationship between theentities represented by the nodes connected by the edge. The domainprocessor 117 can process the text, phrases, or other entities of thedata source with the NLP component 114 to generate the knowledge graph126.

The domain processor 117 can generate a knowledge graph 126 based on theentities of a data source. The data source can be related to orassociated with a digital component. For example, the digital componentcan be third-party content that is displayed with the first-partycontent of a web page. The digital component can include a link to alanding page, web site, or other data source. The domain processor 117can generate a knowledge graph 126 of the data source (e.g., the landingpage) to which the digital component links. The domain processor 117 canstore the knowledge graph 126 in the data repository 124 in associationwith an indication of the data source. The domain processor 117 cangenerate a different knowledge graph 126 for each of the digitalcomponents transmitted to the client computing device 104 for rendering.

The knowledge graph 126 for a digital component can be generated fromonly first order data. For example, the knowledge graph 126 can begenerated based only on the entities and other data included in the datasource 135 that is associated with the digital component. The knowledgegraph 126 for a digital component can be generated from first and secondorder data. First order data can be the entities and other data includedin the data source 135 that is associated with the digital component.Second order data can be entities and other data that are associatedwith different data sources 135 or web searches. The domain processor117 can assign different weighting factors to the first order data andthe second order data. For example, the entities from the first orderdata can have a relatively larger effect on the edge strength betweentwo nodes when compared to the entities from second order data.

The digital component provider device 106 can transmit digitalcomponents to the data processing system 102. The data processing system102 can store the digital components as content data 132. The digitalcomponents can be used as third-party content for websites. The digitalcomponents can include a uniform resource locator or link to a landingpage or other data source 135. When the data processing system 102receives a digital component from a digital component provider device106, the domain processor 117 can generate a knowledge graph 126 for thedata source 135 linked to by the digital component. The domain processor117 can store the knowledge graph 126 in the data repository 124 inassociation with an indication of the data source or the digitalcomponent.

The domain processor 117 can generate and include an interface to theknowledge graph 126 into the digital component. The interface can be alink or deep link that causes an instance of the local digital assistant134 to execute on the client computing device 104. For example, theclient computing device 104 can receive the digital component inresponse to a request. Rendering the digital component can cause theclient computing device 104 to launch or execute an instance of thelocal digital assistant 134. Responses to requests presented to therendered digital component can be generated based on the knowledge graph126.

The data processing system 102 can execute or run an instance of thedigital component selector 120. The digital component selector 120 canselect a digital component that includes text, strings, characters,video files, image files, or audio files that can be processed by theclient computing device 104 and presented to the user via the display144 or the transducer 140 (e.g., speaker).

The digital component selector 120 can select a digital component thatis responsive or associated with the request identified by the NLPcomponent 114 in the input audio signal. The digital component selector120 can select which digital component provider device 106 should or canfulfill the request and can forward the request to the digital componentprovider device 106. For example, the data processing system 102 caninitiate a session between the digital component provider device 106 andthe client computing device 104 to enable the digital component providerdevice 106 to transmit the digital component to the client computingdevice 104. The digital component selector 120 can request a digitalcomponent from the digital component provider device 106. The digitalcomponent provider device 106 can provide digital components to the dataprocessing system 102, which can store the digital components in thedata repository 124. Responsive to a request for a digital component,the digital component selector 120 can retrieve the digital componentfrom the data repository 124.

The digital component selector 120 can select multiple digitalcomponents via a real-time content selection process. The digitalcomponent selector 120 can score and rank the digital components andselect a digital component from a plurality of digital components basedon the score or rank of the digital components. The digital componentselector 120 can select one or more additional digital components thatare transmitted to a second client computing device 104 based on aninput audio signal (or keywords and requests contained therein). Thedigital component selector 120 can select additional digital components(e.g., ads) that are associated with a different digital componentprovider device 106.

The digital component selector 120 can provide the digital componentselected in response to the request identified in the input signal tothe computing device 104, or local digital assistant 134, or applicationexecuting on the computing device 104 for presentation. Thus, thedigital component selector 120 can receive the content request from theclient computing device 104, select, responsive to the content request,a digital component, and transmit, to the client computing device 104,the digital component for presentation. The digital component selector120 can transmit, to the local digital assistant 134, the selecteddigital component for presentation by the local digital assistant 134itself or a third-party application executed by the client computingdevice 104. For example, the local digital assistant 134 can play oroutput an audio signal corresponding to the selected digital component.

The data repository 124 store content data 132 that can include, forexample, digital components provided by a digital component providerdevice 106 or obtained or determined by the data processing system 102to facilitate content selection. The content data 132 can include, forexample, digital components (or a digital component object) that caninclude, for example, a content item, an online document, audio, images,video, multimedia content, or third-party content. The content data 132can include digital components, data, or information provided by theclient computing devices 104 (or end user thereof). For example, thecontent data 132 can include user preferences, user information storedby the user, or data from prior input audio signals.

FIG. 2 illustrates a block diagram of the system 100 generating andusing example knowledge graphs 126 to generate responses. As illustratedin FIG. 2, the system 100 includes a client computing device 104. Theclient computing device 104 is in communication with the data processingsystem 102 and the digital component provider device 106. The digitalcomponent provider device 106 can host, serve, or otherwise beassociated with a data source 135. As illustrated in FIG. 2, the datasource 135 can be a landing page. The landing page can be a website thatis associated with a digital component. For example, the digitalcomponent can include a link to the landing page. The digital componentcan be third-party content that is presented on a website withfirst-party content. In this example, the digital component can be an adfor an ACME coffee maker, and the data source 135 can be a landing pagefor the ACME coffee maker.

The digital component provider device 106 can provide the digitalcomponent (in this example for the ACME coffee maker) to the dataprocessing system 102. The digital component selector 120 can receivethe digital component and store the digital component in the datarepository 124. The data processing system 102 can generate a knowledgegraph 126 based on the digital component. The data processing system 102can generate the knowledge graph 126 based on the data source 135associated with the digital component. For example, the domain processor117 can process, via the NLP component 114, the text or other content ofthe landing page associated with the digital component. The dataprocessing system 102 can store the knowledge graph 126 in the datarepository 124. The digital component provider device 106 can generatethe knowledge graph 126 associated with the digital component andprovide the knowledge graph 126 to the data processing system 102 withthe digital component.

In the example illustrated in FIG. 2, the client computing device 104can be a mobile phone. The client computing device 104 can receive aninput signal 200. The client computing device 104 can receive the inputsignal as an input audio signal. For example, the user can speak arequest to the client computing device 104. An instance of the NLPcomponent 114 (executed at the client computing device 104 or dataprocessing system 102) can parse the input signal to determine the textof the input signal. The client computing device 104 can receive theinput signal via a keyboard. For example, the user can type the request.The NLP component 114 can parse the input signal to identify a request,such as a search request, in the input signal 200. As illustrated inFIG. 2, the NLP component 114 can parse the input signal 200 to identifythe search request as “where is the nearest coffee shop?”

The client computing device 104 can transmit the input signal to thedata processing system 102. The data processing system 102 can select aresponse to the search request parsed from the input signal. The dataprocessing system 102 can transmit the search response to the clientcomputing device 104 for rendering. As illustrated in FIG. 2, theresponse can be rendered to the user as a search response 202. Thesearch response 202 can be rendered as a text, images, video, audio, orany combination thereof. For example, as illustrated in FIG. 2, thesearch response 202 is rendered as text. When the client computingdevice 104 is a speaker-based digital assistant, the search response 202can be rendered as an audio output signal.

Based on the search request, the digital component selector 120 of thedata processing system 102 can select a digital component 204. The dataprocessing system 102 can transmit the digital component 204 to theclient computing device 104. The client computing device 104 can renderthe digital component 204 to display or present the digital component204 to the user with the search response 202. The digital component 204can be included in another digital component 206. The digital component206 can be executed by the local digital assistant 134.

The digital component 206 can include an interface 208 to the knowledgegraph 126. The interface 208 can be configured to receive input signals,such as text or audio-based input signals. The interface 208 can bereferred to as an input interface 208. The digital components 204 or 206can include a link to the landing page 135. Selecting the digitalcomponents 204 or 206 can cause the client computing device 104 toactivate a web browser that loads the address identified in the link forthe landing page 135.

The interface 208 can accept input signals, such as audio-based ortext-based input signals. The client computing device 104 can receivethe input signal, via the interface 208, and transmit the input signalto the data processing system 102. To receive the input signal, theinterface 208 may be activated by a user. For example, the user canselect, click, or tap on the digital component 206 or the interface 208to activate the interface 208 such that the interface 208 begins toreceive input signals. Activating the interface 208 can cause inputsignals to go to the interface 208 rather than the interface throughwith the input signal 200 was received. The local digital assistant 134can determine whether an input signal should be provided to theinterface 208 or provided to the data processing system 102 (as inputsignal 200 was). For example, the local digital assistant 134 canprocess the input signal with a local instance of the NLP component 114and provide the input signal to the interface 208 based on one or moreterms parsed from the input signal and based on the context or keywordsof the digital component 204.

The data processing system 102, when receiving input signals via theinterface 208, can generate a response based on the knowledge graph 126.For example, the NLP component 114 can process the input signal to parsea request in the input signal. The request can be a request foradditional information related to the digital component 204. Forexample, in the example illustrated in FIG. 2, the request can be foradditional information regarding the ACME coffee maker. The dataprocessing system 102 can use the knowledge graph 126 that is associatedwith the digital component 204 to generate a response to the request.For example, the knowledge graph 126 can be based on the landing page135 that is associated with the digital component 204 such that theresponse provides responses specific to the entities, text, and otherdata contained within the landing page 135. For example, the requestreceived via the interface 208 can be “what is the cost of the coffeemaker?” The data processing system 102 can generate a response to therequest using the knowledge graph 126. The response can include the costof the coffee maker as indicated on the landing page 135.

The digital component 206 can be transmitted to the client computingdevice 104 in response to a search request. The digital component 206can be transmitted to the client computing device 104 in response to arequest for third-party content. For example, the client computingdevice 104 can load a web page that includes first-party content. Theweb page can include slots for third-party content. The web page caninclude scripts or other processor executable instruction that, whenexecuted, cause the client computing device 104 to request third-partycontent from the data processing system 102. The data processing system102, via the digital component selector 120, can select the digitalcomponent 206 based on entities, keywords, content, or data associatedwith the web page's first-party content. The data processing system 102can generate the digital component 206 to include the digital component204 from the digital component provider device 106 and the interface208. The interface 208 can be processor executable instructions thatcause the client computing device 104 to launch an instance of the localdigital assistant. While the user views the web page that includes thefirst-party content and the digital component 206, the user can inputsignals into the interface 208. The interface 208 can transmit the inputsignals to the data processing system 102. The interface 208 cantransmit the input signals to the data processing system 102 withoutcausing the browser to leave the web page. This can enable the user togather or request additional information about the digital component 204without leaving the web page.

The client computing device 104 can include a voice-only interface. Forexample, rather than displaying the search response 202, the clientcomputing device 104 can render the search response 202 as an outputaudio file. The digital components 206 and 204 can also be rendered tothe user as an output audio signal, for example, after the rendering ofthe search response 202. The user can speak to the client computingdevice 104 to provide input signals to the interface 208.

FIG. 3 illustrates a block diagram of an example method 300 to generateresponses based on domain-specific natural language processing models ina networked system. The method 300 can include receiving an input audiosignal (ACT 302). The method 300 can include parsing the input audiosignal (ACT 304). The method 300 can include selecting a search result(ACT 306). The method 300 can include selecting a digital component (ACT308). The method 300 can include transmitting the search result anddigital component (ACT 310). The method 300 can include receiving arequest (ACT 312). The method 300 can include selecting a response (ACT314). The method 300 can include transmitting the response (ACT 316).

The method 300 can include can include receiving an input signal (ACT302). The method can include receiving, by an NLP component executed bya data processing system, the input signal. The input signal can be aninput audio signal that is detected by a sensor at a first client deviceand transmitted to the data processing system. The sensor can be amicrophone of the first client device. The input signal can be a typedrequest. A digital assistant component executed at least partially by adata processing system that includes one or more processors and memorycan receive the input signal. The input signal can include aconversation facilitated by a digital assistant. The conversation caninclude one or more inputs and outputs. The conversation can be audiobased, text based, or a combination of audio and text. The input audiosignal can include text input, or other types of input that can provideconversational information. The data processing system can receive theinput for a session corresponding to the conversation.

The method 300 can include parsing the input signal (ACT 304). The NLPcomponent of the data processing system can parse the input signal toidentify a request. The request can be a search request. The request canbe an intent or request for data, information, web pages, or a search.The NLP component can identify one or more entities, such as terms orphrases, in the request.

The method 300 can include selecting a search result (ACT 306). The dataprocessing system can select a search result that is based at least onthe search request parsed from the input signal. The data processingsystem can include or can interface with a search engine or search dataprocessing system that can select one or more search results and providethe search results to the client computing device.

The method 300 can include selecting a digital component (ACT 308). Thedata processing system can select the digital component based on thesearch request. For example, digital component provider devices canprovide the data processing system candidate digital components. Thedigital component provider device can associate or label the candidatedigital components with keywords. The digital component selector canselect one or more digital components from the candidate digitalcomponents based on a similarity between the digital components'keywords and entities (e.g., terms) identified in the request.

The similarity can be a match. For example, the data processing systemcan select a digital component having a keyword that is present as anentity in the request. For example, and referring to FIG. 2, the digitalcomponent 204 can be labeled with the keyword “coffee.” The digitalcomponent selector can select the digital component 204 because the term“coffee” is present in the input signal 200.

The similarity can be based on a semantic relationship. For example, thesearch can be for vacation rentals and the data processing system canmatch the search to digital components that include the keyword “flightreservations” because the phrase “flight reservations” can besemantically related to vacation rentals.

The digital component can be associated with a data source, such as alanding page or other website. A digital component can be associatedwith a data source when the digital component includes a link to thedata source. The digital component can include a link such that, whenthe link is activated by a client computing device, the link causes aweb browser executed by the client computing device to retrieve the datasource. The data processing system can generate a knowledge graph basedon the data source. For example, the knowledge graph can be generatedfrom the terms, phrases, or other entities included in the data source.A digital component provider device can generate the knowledge graph andprovide the knowledge graph to the data processing system with thecandidate digital components.

The method 300 can include transmitting the result and digital component(ACT 310). The data processing system can transmit the search result andthe digital component to the client computing device that transmittedthe input signal to the data processing system. Once the clientcomputing device receives the result and the digital component, theclient computing device can render the result and the digital component.Rendering the digital component can cause the activation or execution ofan instance of a local digital assistant at the client computing device.The local digital assistant can render or otherwise display the digitalcomponent. The local digital assistant can render or otherwise displaythe result.

The digital component can include an interface to the knowledge graph.For example, the digital component, when rendered, can present an inputinterface, such as a graphical interface, to the user. Via the inputinterface, the user can input requests. The requests can be transmittedto the data processing system via the digital component.

The method 300 can include receiving the request (ACT 312). The requestcan be within an input signal. The input signal can be an audio-based ora text-based input signal. For example, the user can speak a question tothe input interface which can be detected by a microphone of the clientcomputing device. The local digital assistant can receive the inputsignal and transmit the input signal to the data processing system. Whenthe input signal is an input audio signal, the NLP component can receivethe input signal and parse the request from the input signal.

The method 300 can include selecting a response (ACT 314). The digitalcomponent selector can select the response to the request included inthe input signal. The data processing system can generate the responsebased on the knowledge graph that is associated with the digitalcomponent that was transmitted to the client computing device at ACT310.

The method 300 can include transmitting the response (ACT 316). The dataprocessing system can transmit the response to the client computingdevice. The data processing system can include the response into adigital component that includes the response and instructions for how torender the response at the client computing device. The client computingdevice can render or display the response within the digital componenttransmitted to the client computing device at ACT 310. Rendering theresponse within the digital component previously transmitted to theclient computing device can enable the response to be presented to theuser without changing or altering content currently displayed to theuser. For example, and also referring to FIG. 2, rendering the resultwithin the digital component 206 can enable the input signal 200 and thesearch response 202 to remain displayed to the user while the userrequests and receives additional information about the digital component204.

FIG. 4 illustrates a block diagram of an example method 400 to generateresponses based on domain-specific natural language processing models ina networked system. The method 400 can include receiving a request (ACT402). The method 400 can include selecting a digital component (ACT404). The method 400 can include transmitting the digital component (ACT406). The method 400 can include receiving an input signal (ACT 408).The method 400 can include parsing the input signal (ACT 410). Themethod 400 can include selecting a response (ACT 412). The method 400can include transmitting the response (ACT 414).

The method 400 can include receiving a request (ACT 402). The requestcan be a request for third-party content. The request can be receivedfrom a client computing device. For example, the client computing devicecan include a web browser. The web browser can receive and render awebsite that includes first-party content. The website can include slotsfor third-party content. The slots can include processor executableinstructions that can cause the web browser to transmit a contentrequest to the data processing system. The request can include contentparameters. The content parameters can be size requirements for thereturned digital component or keywords that the data processing systemcan use to select the digital component.

The method 400 can include selecting a digital component (ACT 404). Thedata processing system can select a digital component from a pluralityof digital components. The digital component selector of the dataprocessing system can select the digital component. The digitalcomponent selector can select multiple digital components via areal-time content selection process. The digital component selector canscore and rank the digital components. The digital component selectorcan select a digital component from a plurality of digital componentsbased on the score and rank of the digital components. For example, thedigital component selector can select the digital component with thehighest rank or score. The data processing system can include thedigital component into another digital component with an interface to aknowledge graph associated with the selected digital component.

The method 400 can include transmitting the digital component (ACT 406).The data processing system can transmit the selected digital componentto the client computing device. The client computing device can receivethe digital component and render the digital component in one of theslots of the web page as third-party content. Rendering the digitalcomponent with the first-party content can present an interface to theknowledge graph associated with the digital component to an end user.For example, the rendering of the digital component can cause the clientcomputing device to launch or execute a local instance of the digitalassistant. The interface can be configured to receive input signals. Theinput signals can be text-based or audio-based. The client computingdevice can transmit the input signal received via the digital componentto the data processing system.

The method 400 can include receiving the input signal (ACT 408). Themethod 400 can include receiving, by an NLP component executed by a dataprocessing system, the input signal. The input signal can be an inputaudio signal that is detected by a sensor at a first client device andtransmitted to the data processing system. The sensor can be amicrophone of the first client device. The input signal can be a typedrequest. A digital assistant component executed at least partially by adata processing system that includes one or more processors and memorycan receive the input signal.

The method 400 can include parsing the input signal (ACT 410). The NLPcomponent of the data processing system can parse the input signal toidentify a request. The request can be a search request. The request canbe an intent or request for data, information, webpages, or a search.The NLP component can identify one or more entities, such as terms orphrases, in the request. For example, the request can be for informationor data related to the digital component provided to the clientcomputing device as third-party content.

The method 400 can include selecting a response (ACT 412). The digitalcomponent from which the input signal was transmitted can be associatedwith a knowledge graph. The knowledge graph can be generated from theterms or entities contained within a data source (e.g., a landing page)that is associated with the digital component. The data processingsystem can select the response based on the knowledge graph associatedwith the digital component. For example, the data processing system'sNLP component can use the knowledge graph to generate the response basedon the entities and data contained in the landing page.

The method 400 can include transmitting the response (ACT 414). The dataprocessing system can transmit the response to the client computingdevice. The client computing device can receive the response and renderthe response within the digital component transmitted to the clientcomputing device at ACT 406. Rendering the response within the digitalcomponent can enable the response to be presented to the user withoutchanging or altering the first-party content presented to the user. Forexample, the user does not need to leave the original website displayingthe first-party content to view or receive the response.

FIG. 5 is a block diagram of an example computer system 500. Thecomputer system or computing device 500 can include, or be used toimplement, the system 100 or its components such as the data processingsystem 102. The data processing system 102 can include an intelligentpersonal assistant or voice-based digital assistant. The computingsystem 500 includes a bus 505 or other communication component forcommunicating information and a processor 510 or processing circuitcoupled to the bus 505 for processing information. The computing system500 can also include one or more processors 510 or processing circuitscoupled to the bus for processing information. The computing system 500also includes main memory 515, such as a random access memory (RAM) orother dynamic storage device, coupled to the bus 505 for storinginformation, and instructions to be executed by the processor 510. Themain memory 515 can be or include the data repository 124. The mainmemory 515 can also be used for storing position information, temporaryvariables, or other intermediate information during execution ofinstructions by the processor 510. The computing system 500 may furtherinclude a read-only memory (ROM) 520 or other static storage devicecoupled to the bus 505 for storing static information and instructionsfor the processor 510. A storage device 525, such as a solid-statedevice, magnetic disk or optical disk, can be coupled to the bus 505 topersistently store information and instructions. The storage device 525can include or be part of the data repository 124.

The computing system 500 may be coupled via the bus 505 to a display535, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 530, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 505 for communicating information and command selections to theprocessor 510. The input device 530 can include a touch screen display535. The input device 530 can also include a cursor control, such as amouse, a trackball, or cursor direction keys for communicating directioninformation and command selections to the processor 510 and forcontrolling cursor movement on the display 535. The display 535 can bepart of the data processing system 102, the client computing device 104or other component of FIG. 1, for example.

The processes, systems and methods described herein can be implementedby the computing system 500 in response to the processor 510 executingan arrangement of instructions contained in main memory 515. Suchinstructions can be read into main memory 515 from anothercomputer-readable medium, such as the storage device 525. Execution ofthe arrangement of instructions contained in main memory 515 causes thecomputing system 500 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory515. Hard-wired circuitry can be used in place of or in combination withsoftware instructions together with the systems and methods describedherein. Systems and methods described herein are not limited to anyspecific combination of hardware circuitry and software.

Although an example computing system has been described in FIG. 5, thesubject matter including the operations described in this specificationcan be implemented in other types of digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them.

For situations in which the systems discussed herein collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's location), or to control whether or how toreceive content from a content server or other data processing systemthat may be more relevant to the user. In addition, certain data may beanonymized in one or more ways before it is stored or used, so thatpersonally identifiable information is removed when generatingparameters. For example, a user's identity may be anonymized so that nopersonally identifiable information can be determined for the user, or auser's geographic location may be generalized where location informationis obtained (such as to a city, postal code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about him or her and usedby the content server.

The subject matter and the operations described in this specificationcan be implemented in digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. The subject matter described in thisspecification can be implemented as one or more computer programs (e.g.,one or more circuits of computer program instructions), encoded on oneor more computer storage media for execution by, or to control theoperation of, data processing apparatuses. Alternatively, or inaddition, the program instructions can be encoded on an artificiallygenerated propagated signal (e.g., a machine-generated electrical,optical, or electromagnetic signal) that is generated to encodeinformation for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. While acomputer storage medium is not a propagated signal, a computer storagemedium can be a source or destination of computer program instructionsencoded in an artificially generated propagated signal. The computerstorage medium can also be, or be included in, one or more separatecomponents or media (e.g., multiple CDs, disks, or other storagedevices). The operations described in this specification can beimplemented as operations performed by a data processing apparatus ondata stored on one or more computer-readable storage devices or receivedfrom other sources.

The terms “data processing system,” “computing device,” “component,” or“data processing apparatus” encompass various apparatuses, devices, andmachines for processing data, including by way of example a programmableprocessor, a computer, a system on a chip, or multiple ones, orcombinations of the foregoing. The apparatus can include special purposelogic circuitry (e.g., an FPGA (field programmable gate array) or anASIC (application specific integrated circuit). The apparatus can alsoinclude, in addition to hardware, code that creates an executionenvironment for the computer program in question (e.g., code thatconstitutes processor firmware, a protocol stack, a database managementsystem, an operating system, a cross-platform runtime environment, avirtual machine, or a combination of one or more of them). The apparatusand execution environment can realize various different computing modelinfrastructures, such as web services, distributed computing and gridcomputing infrastructures. For example, the interface 110, digitalcomponent selector 120, domain processor 117, or NLP component 114 andother data processing system 102 components can include or share one ormore data processing apparatuses, systems, computing devices, orprocessors.

A computer program (also known as a program, software, softwareapplication, app, script, or code) can be written in any form ofprogramming language, including compiled or interpreted languages,declarative or procedural languages, and can be deployed in any form,including as a stand-alone program or as a module, component,subroutine, object, or other unit suitable for use in a computingenvironment. A computer program can correspond to a file in a filesystem. A computer program can be stored in a portion of a file thatholds other programs or data (e.g., one or more scripts stored in amarkup language document) in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs (e.g., components of the data processing system 102)to perform actions by operating on input data and generating output. Theprocesses and logic flows can also be performed by, and apparatuses canalso be implemented as, special purpose logic circuitry (e.g., an FPGAor an ASIC). Devices suitable for storing computer program instructionsand data include all forms of non-volatile memory, media, and memorydevices, including by way of example semiconductor memory devices (e.g.,EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internalhard disks or removable disks); magneto optical disks; and CD ROM andDVD-ROM disks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

The subject matter described herein can be implemented in a computingsystem that includes a back end component (e.g., as a data server), orthat includes a middleware component (e.g., an application server), orthat includes a front end component (e.g., a client computer having agraphical user interface or a web browser through which a user caninteract with an implementation of the subject matter described in thisspecification), or a combination of one or more such back end,middleware, or front end components. The components of the system can beinterconnected by any form or medium of digital data communication(e.g., a communication network). Examples of communication networksinclude a local area network (“LAN”) and a wide area network (“WAN”), aninter-network (e.g., the Internet), and peer-to-peer networks (e.g., adhoc peer-to-peer networks).

The computing system such as system 100 or system 500 can includeclients and servers. A client and server are generally remote from eachother and typically interact through a communication network (e.g., thenetwork 105). The relationship of a client and server arises by virtueof computer programs running on the respective computers and having aclient-server relationship to each other. In some implementations, aserver transmits data (e.g., data packets representing a digitalcomponent) to a client device (e.g., for purposes of displaying data toand receiving user input from a user interacting with the clientdevice). Data generated at the client device (e.g., a result of the userinteraction) can be received from the client device at the server (e.g.,received by the data processing system 102 from the client computingdevice 104 or the digital component provider device 106).

While operations are depicted in the drawings in a particular order,such operations are not required to be performed in the particular ordershown or in sequential order, and all illustrated operations are notrequired to be performed. Actions described herein can be performed in adifferent order.

The separation of various system components does not require separationin all implementations, and the described program components can beincluded in a single hardware or software product. For example, the NLPcomponent 114, domain processor 117, or the digital component selector120, can be a single component, app, or program, or a logic devicehaving one or more processing circuits, or part of one or more serversof the data processing system 102.

Having now described some illustrative implementations, it is apparentthat the foregoing is illustrative and not limiting, having beenpresented by way of example. In particular, although many of theexamples presented herein involve specific combinations of method actsor system elements, those acts and those elements may be combined inother ways to accomplish the same objectives. Acts, elements andfeatures discussed in connection with one implementation are notintended to be excluded from a similar role in other implementations orimplementations.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including,” “comprising,” “having,” “containing,” “involving,”“characterized by,” “characterized in that,” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular may also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein mayalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element may include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein may be combined with any otherimplementation or embodiment, and references to “an implementation,”“some implementations,” “one implementation” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described in connectionwith the implementation may be included in at least one implementationor embodiment. Such terms as used herein are not necessarily allreferring to the same implementation. Any implementation may be combinedwith any other implementation, inclusively or exclusively, in any mannerconsistent with the aspects and implementations disclosed herein.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall the described terms. For example, a reference to “at least one of‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and‘B’. Such references used in conjunction with “comprising” or other openterminology can include additional items.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded to increase the intelligibility of the drawings, detaileddescription, and claims. Accordingly, neither the reference signs northeir absence has any limiting effect on the scope of any claimelements.

The systems and methods described herein may be embodied in otherspecific forms without departing from the characteristics thereof. Forexample, the computing device 104 can generate the packaged data objectand forward it to the third-party application when launching theapplication. The foregoing implementations are illustrative rather thanlimiting of the described systems and methods. Scope of the systems andmethods described herein is thus indicated by the appended claims,rather than the foregoing description, and changes that come within themeaning and range of equivalency of the claims are embraced therein.

What is claimed:
 1. A system to generate responses based ondomain-specific natural language processing models in a networkedsystem, comprising: a data processing system comprising one or moreprocessors and memory, the one or more processors execute a naturallanguage processor component and a digital component selector to:receive, by the natural language processor component and via aninterface of the data processing system, an input audio signal detectedby a sensor at a first client device; parse, by the natural languageprocessor component, the input audio signal to identify a first searchrequest in the input audio signal; select, by a digital componentselector executed by the data processing system, a search result basedon the at least the first search request; select, by the digitalcomponent selector, a digital component based on the first searchrequest, the digital component comprising a link to a data sourcecomprising a plurality of entities and associated with a knowledge graphrepresenting a relationship between the plurality of entities of thedata source, the digital component comprising an input interface torequest responses based on the knowledge graph; transmit, by theinterface, the digital component comprising the link to the data sourceand associated with the knowledge graph to the first client device withthe search result; receive, by the interface and via the input interfaceof the digital component rendered by the client device, a second searchrequest; select, by the digital component selector and based on theknowledge graph, a response based to the second search request; andtransmit, by the interface, the response to the first client device torender the response within the digital component.
 2. The system of claim1, comprising: a domain processor executed by the data processing systemto generate the knowledge graph based on the plurality of entities ofthe data source.
 3. The system of claim 1, wherein the knowledge graphrepresents semantic relationships between the plurality of entities ofthe data source.
 4. The system of claim 1, wherein the data sourcecomprises a landing page for the digital component.
 5. The system ofclaim 1, comprising: the natural language processor component to receivethe second search request as a second input audio signal detected by thesensor of the first client device via the input interface of the digitalcomponent; the natural language processor component to parse the secondinput audio signal to identify a request in the second input audiosignal; a domain processor to generate a second digital componentcomprising the response to the request in the second input audio signalbased at least on the knowledge graph; and the interface to transmit thesecond digital component to the first client device.
 6. The system ofclaim 5, comprising: the digital component selector to select a contentprovider based on the request in the second input audio signal; thedigital component selector to include a link to the content provider inthe digital component.
 7. The system of claim 1, comprising: the digitalcomponent selector to select a second data source based on the secondsearch request, the second data source comprising a second plurality ofentities; and the domain processor to generate the knowledge graphrepresenting semantic relationships between a plurality of entities ofthe data source and the second plurality of entities of the second datasource.
 8. The system of claim 1, wherein knowledge graph comprisesnodes representing each of a plurality of entities of the data sourceand edges representing a relationship between the nodes.
 9. The systemof claim 8, comprising: a domain processor to calculate a weight foreach of the edges representing the relationship between nodes.
 10. Thesystem of claim 1, comprising: the natural language processor componentto receive a second input signal detected received by the first clientdevice; the natural language processor component to parse the secondinput signal to identify a request in the second input signal; a domainprocessor to generate a second digital component comprising a responseto the request in the second input signal based at least on theknowledge graph; and the interface to transmit the second digitalcomponent to the first client device in response to the request in thesecond input signal.
 11. The system of claim 1, comprising: the digitalcomponent selector to receive a content request from the first clientdevice; the digital component selector to select a second digitalcomponent based on the content request, the digital component associatedwith a second data source and a second knowledge graph based on thesecond data source; the digital component selector transmits the seconddigital component to the first client device to render the digitalcomponent in a content slot; the natural language processor component toreceive a second input audio signal detected by a sensor at the firstclient device; the natural language processor component to parse theinput audio signal to identify a second request in the input audiosignal; the natural language processor component to select a response tothe second request based on the second knowledge graph; and theinterface to transmit the response to the client computing device.
 12. Amethod to generate responses based on domain-specific natural languageprocessing models in a networked system comprising: receiving, by anatural language processor component executed by a data processingsystem and via an interface of the data processing system, an inputaudio signal detected by a sensor at a first client device; parsing, bythe natural language processor component, the input audio signal toidentify a first search request in the input audio signal; selecting, bya digital component selector executed by the data processing system, asearch result based on the at least the first search request; selecting,by the digital component selector, a digital component based on thefirst search request, the digital component associated with a datasource and a knowledge graph based on the data source; transmitting, bythe digital component selector, the digital component associated withthe data source and the knowledge graph based on the data source to thefirst client device with the search result.
 13. The method of claim 12,comprising: generating, by a domain processor executed by the dataprocessing system, the knowledge graph based on one or more entitiescontained in the data source.
 14. The method of claim 12, wherein theknowledge graph represents semantic relationships between a plurality ofentities of the data source.
 15. The method of claim 12, wherein thedata source comprises a landing page for the digital component.
 16. Themethod of claim 12, comprising: receiving, by the natural languageprocessor component, a second input audio signal detected by the sensorof the first client device via the digital component; parsing, by thenatural language processor component, the second input audio signal toidentify a request in the second input audio signal; generating, by adomain processor, a second digital component comprising a response tothe request in the second input audio signal based at least on theknowledge graph; and transmitting, by the interface, the second digitalcomponent to the first client device.
 17. The method of claim 16,comprising: selecting, by the digital component selector, a contentprovider based on the request in the second input audio signal;generating, by the digital component selector, a link to the contentprovider; and including the link in the digital component.
 18. Themethod of claim 16, comprising: selecting, by the digital componentselector, a second data source based on the request, the second datasource comprising a second plurality of entities; and generating, by thedomain processor, the knowledge graph representing semanticrelationships between a plurality of entities of the data source and thesecond plurality of entities of the second data source.
 19. The methodof claim 12, comprising: receiving, by the natural language processorcomponent and via the digital component, a second input signal detectedreceived by the first client device; parsing, by the natural languageprocessor component, the second input signal to identify a request inthe second input signal; generating, by a domain processor, a seconddigital component comprising a response to the request in the secondinput signal based at least on the knowledge graph; and transmitting, bythe domain processor, the second digital component to the first clientdevice in response to the request in the second input signal.
 20. Themethod of claim 12, comprising: receiving, by a digital componentselector executed by the data processing system, a content request fromthe first client device; selecting, by the digital component selector, asecond digital component based on the content request, the digitalcomponent associated with a second data source and a second knowledgegraph based on the second data source; transmitting, by the digitalcomponent selector, the second digital component to the first clientdevice to render the digital component in a content slot; receiving, bythe natural language processor component, a second input audio signaldetected by a sensor at the first client device; parsing, by the naturallanguage processor component, the input audio signal to identify asecond request in the input audio signal; selecting, by the naturallanguage processor component, a response to the second request based onthe second knowledge graph; and transmitting, by the interface, theresponse to the client computing device.